Uso de padrões em projeto arquitetural baseado em busca de linha de produto de Software

Orientadora : Profª. Drª. Silvia Regina VergilioCo-orientadora : Profª. Drª. Thelma Elita Colanzi LopesDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa: Curitiba, 23/10/2014Inclui referênciasResumo: Padrões de projeto visam a melhorar o entendimento e o reúso de arquiteturas de software. No projeto baseado em busca eles têm sido aplicados com sucesso por meio de operadores de mutação em processos evolutivos. No contexto de Arquiteturas de Linha de Produtos (ALPs), alguns trabalhos têm aplicado padrões de projeto manualmente, mas não existem abordagens baseadas em busca que considerem o uso destes padrões. Tornar este uso possível é o objetivo deste trabalho, que introduz uma forma automática para aplicação de padrões de projeto por meio de um operador de mutação na abordagem Multi-objective Optimization Approach for PLA Design (MOA4PLA). A ideia é que esta aplicação não gere anomalias na arquitetura e garanta que estes padrões sejam aplicados em escopos realmente propícios para suas aplicações. Para isso, foi realizada uma análise de viabilidade para determinar quais padrões do catálogo GoF (Gang of Four) são aplicáveis no contexto da MOA4PLA. Um operador de mutação é proposto para ser utilizado neste contexto, de modo a aplicar estes padrões de projeto em ALPs durante o processo evolutivo utilizando métodos de verificação de escopos e aplicação de padrões. O operador de mutação foi implementado no módulo OPLA-Patterns da ferramenta OPLA-Tool, que dá suporte a abordagem MOA4PLA. Experimentos foram configurados e executados em ALPs reais para avaliar quantitativamente e qualitativamente os resultados obtidos. Os resultados mostram que a aplicação de padrões de projeto permite a obtenção de arquiteturas com melhores valores em métricas de software resultando em uma maior diversidade de soluções para que o arquiteto possa escolher qual delas mais se adequa aos seus objetivos. Portanto, aplicar padrões de projeto por meio do operador proposto contribui positivamente para o projeto de ALP.Abstract: Design patterns aim at improving the understanding and reuse of software architectures. In the search-based design they have been successfully applied by mutation operators in the evolutionary process. In the software Product Line Architecture (PLA) context, some works have manually applied design patterns, but there are no search-based approaches that take into account the use of these patterns. To make this use possible is the goal of this work, which introduces an automated way for the application of design patterns through a mutation operator in the MOA4PLA approach (Multi-objective Optimization Approach for PLA Design). The goal is to avoid the introduction of architectural anomalies and to ensure that these patterns are applied only in feasible scopes. To this end, a feasibility analysis was conducted to determine which patterns of the GoF (Gang of Four) catalog could be applied in the context of MOA4PLA. A mutation operator is proposed in order to apply feasible design patterns in PLAs during the evolutionary process. The operator uses scope verification and design patterns application methods. The mutation operator was implemented in the OPLA-Patterns module of OPLA-Tool, which supports the MOA4PLA approach. Experiments were configured and executed in real PLAs to quantitatively and qualitatively evaluate the results. The results showed that the application of design patterns allows the generation of architectures with better values of the software metrics. A greater diversity of solutions was obtained, then the architect can choose which one best fits his/her objectives. Therefore, the application of design patterns using the proposed operator contributes positively to the PLA design

MEG: Multi-objective Ensemble Generation for Software Defect Prediction

Background: Defect Prediction research aims at assisting software engineers in the early identification of software defect during the development process. A variety of automated approaches, ranging from traditional classification models to more sophisticated learning approaches, have been explored to this end. Among these, recent studies have proposed the use of ensemble prediction models (i.e., aggregation of multiple base classifiers) to build more robust defect prediction models. / Aims: In this paper, we introduce a novel approach based on multi-objective evolutionary search to automatically generate defect prediction ensembles. Our proposal is not only novel with respect to the more general area of evolutionary generation of ensembles, but it also advances the state-of-the-art in the use of ensemble in defect prediction. / Method: We assess the effectiveness of our approach, dubbed as Multi-objective Ensemble Generation (MEG), by empirically benchmarking it with respect to the most related proposals we found in the literature on defect prediction ensembles and on multi-objective evolutionary ensembles (which, to the best of our knowledge, had never been previously applied to tackle defect prediction). / Result: Our results show that MEG is able to generate ensembles which produce similar or more accurate predictions than those achieved by all the other approaches considered in 73% of the cases (with favourable large effect sizes in 80% of them). / Conclusions: MEG is not only able to generate ensembles that yield more accurate defect predictions with respect to the benchmarks considered, but it also does it automatically, thus relieving the engineers from the burden of manual design and experimentation

Fuzzing vs SBST : intersections & differences

Search-Based Software Testing (SBST) is the application of SBSE for solving hard software testing problems. SBST has been the subject of discussion of our International SBST Workshop, colocated with the International Conference on Software Engineering (ICSE). In 2022 we hosted the 15th edition of our SBST Workshop, which brought together researchers and industrial practitioners to encourage the use of search-based and fuzz testing techniques and tools for addressing software engineering-specific challenges. In this 2022 edition, SBST held, among other exciting events, a discussion panel on the similarities and differences between SBST and Fuzzing. As it implies, the goal of the panel was to have a common ground for discussion on the main similarities and differences between the Fuzzing and SBST fields, focusing on how both communities can collaborate to advance the stateof-the-art on automated testing. This strong panel composed of researchers from both academia and industry was the highlight of SBST’22 and allowed the chairs of the workshop to make substantial changes for 2023. In this paper, we present the main take away messages from that seminal panel and highlight exciting new challenges in the field

Sentinel: A Hyper-Heuristic for the Generation of Mutant Reduction Strategies

Mutation testing is an effective approach to evaluate and strengthen software test suites, but its adoption is currently limited by the mutants' execution computational cost. Several strategies have been proposed to reduce this cost (a.k.a. mutation cost reduction strategies), however none of them has proven to be effective for all scenarios since they often need an ad-hoc manual selection and configuration depending on the software under test (SUT). In this paper, we propose a novel multi-objective evolutionary hyper-heuristic approach, dubbed Sentinel, to automate the generation of optimal cost reduction strategies for every new SUT. We evaluate Sentinel by carrying out a thorough empirical study involving 40 releases of 10 open-source real-world software systems and both baseline and state-of-the-art strategies as a benchmark. We execute a total of 4,800 experiments, and evaluate their results with both quality indicators and statistical significance tests, following the most recent best practice in the literature. The results show that strategies generated by Sentinel outperform the baseline strategies in 95% of the cases always with large effect sizes. They also obtain statistically significantly better results than state-of-the-art strategies in 88% of the cases, with large effect sizes for 95% of them. Also, our study reveals that the mutation strategies generated by Sentinel for a given software version can be used without any loss in quality for subsequently developed versions in 95% of the cases. These results show that Sentinel is able to automatically generate mutation strategies that reduce mutation testing cost without affecting its testing effectiveness (i.e. mutation score), thus taking off from the tester's shoulders the burden of manually selecting and configuring strategies for each SUT.Comment: in IEEE Transactions on Software Engineerin

Comparing Fuzzers on a Level Playing Field with FuzzBench

Fuzzing is a testing approach commonly used in industry to discover bugs in a given software under test (SUT). It consists of running a SUT iteratively with randomly generated (or mutated) inputs, in order to find as many as possible inputs that make the SUT crash. Many fuzzers have been proposed to date, however no consensus has been reached on how to properly evaluate and compare fuzzers. In this work we evaluate and compare nine prominent fuzzers by carrying out a thorough empirical study based on an open-source framework developed by Google, namely FuzzBench, and a manually curated benchmark suite of 12 real-world software systems. The results show that honggfuzz and AFL++ are, in that order, the best choices in terms of general purpose fuzzing effectiveness. The results also show that none of the fuzzers outperforms the others in terms of efficiency across all considered metrics, that no particular bug affinity is found for any fuzzer, and that the correlation found between coverage and number of bugs depends more on the SUT rather than on the fuzzer used

Applying design patterns in the search-based optimization of software product line architectures

The design of the product line architecture (PLA) is a difficult activity that can benefit from the application of design patterns and from the use of a search-based optimization approach, which is generally guided by different objectives related, for instance, to cohesion, coupling and PLA extensibility. The use of design patterns for PLAs is a recent research field, not completely explored yet. Some works apply the patterns manually and for a specific domain. Approaches to search-based PLA design do not consider the usage of these patterns. To allow such use, this paper introduces a mutation operator named “Pattern-Driven Mutation Operator” that includes methods to automatically identify suitable scopes and apply the patterns Strategy, Bridge and Mediator with the search-based approach multi-objective optimization approach for PLA. A metamodel is proposed to represent and identify suitable scopes to receive each one of the patterns, avoiding the introduction of architectural anomalies. Empirical results are also presented, showing evidences that the use of the proposed operator produces a greater diversity of solutions and improves the quality of the PLAs obtained in the search-based optimization process, regarding the values of software metrics

Sentinel : a multi-objective hyper-heuristic approach for the automatic generation of mutant reduction strategies

Orientadora: Dr. Silvia Regina VergilioCoorientadora: Dr. Federica SarroTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa : Curitiba, 03/12/2018Inclui referências: p.83-90Área de concentração: Ciência da ComputaçãoResumo: O Teste de Mutação utiliza programas mutantes gerados com pequenas modificações no programa em teste. Tais modificações são produzidas por operadores de mutação que descrevem possíveis defeitos. O objetivo é matar os mutantes com casos de teste que produzam diferentes saídas para os mutantes e para o programa original. Ao final, uma medida de avaliação do teste, o escore de mutação, é dada considerando o número de mutantes mortos em relação ao número de mutantes gerados. O teste de mutação é bastante eficaz em revelar defeitos, entretanto, possui um alto custo computacional relacionado à execução dos mutantes. Diferentes estratégias existem para reduzir o custo do teste de mutação. Por exemplo, um subconjunto de mutantes pode ser selecionado aleatoriamente, ou pode-se utilizar um conjunto menor de operadores, ou ainda algoritmos de busca para selecionar os melhores mutantes ou operadores sem reduzir o escore global. Experimentos reportados na literatura mostram que nenhuma estratégia pode ser considerada a melhor em todos os contextos. A determinação das melhores estratégias e suas combinações de parâmetros só é possível com a condução de experimentos comparando estratégias em cenários específicos, o que pode elevar o custo do teste. O uso de uma estratégia inadequada pode comprometer a eficácia do teste e implica em uma redução de custo insatisfatória. Uma abordagem automática para selecionar e configurar estratégias pode auxiliar o testador nesta tarefa. Hiper-heurísticas são opções viáveis para este propósito, uma vez que estas são utilizadas para selecionar ou gerar boas heurísticas (estratégias) ao invés de tentar resolver o problema diretamente. Hiper-heurísticas vêm sendo estudadas na literatura, e bons resultados foram obtidos na área de Engenharia de Software Baseada em Busca. Todavia, poucos trabalhos investigam a aplicação de hiper-heurísticas no teste de mutação, e nenhum deles auxilia na geração ou configuração de estratégias de teste de mutação. Diante disto, este trabalho propõe uma abordagem baseada em hiper-heurísticas chamada Sentinel. Essa abordagem tem como objetivo principal gerar automaticamente estratégias de redução de mutantes de modo a reduzir o custo do teste de mutação sem comprometer a eficácia do teste em termos de escore de mutação e a habilidade de revelar defeitos. Sentinel utiliza elementos provenientes de diversos tipos de estratégias de redução de mutantes e os combina de modo a gerar novas estratégias potencialmente mais efetivas. A ideia é que o testador execute Sentinel, reutilize as estratégias geradas automaticamente e consequentemente não necessite escolher e nem configurar as estratégias manualmente. Isso facilita a aplicação do teste de mutação e reduz assim o custo da atividade de teste como um todo. Para avaliar a viabilidade de Sentinel, múltiplos experimentos foram executados. Os experimentos foram conduzidos em 10 diferentes sistemas com 4 versões cada, totalizando 40 sistemas. As estratégias geradas por Sentinel se mostraram significativamente mais efetivas do que estratégias convencionais da literatura. Além disso, elas mantiveram os seus bons resultados em novas versões dos softwares, o que demonstrou a sua reusabilidade. Palavras-chave: hiper-heurística, teste de software, geração de estratégias, redução de mutantes, evolução gramatical, engenharia de software baseada em busca, otimização multiobjetivo.Abstract: Mutation Testing uses mutant programs generated with small modifications in the program under test. Such modifications are produced by mutation operators that describe possible faults. The objective is to kill the mutants with test cases that produce different outputs for the mutants and the original program. In the end, a test assessment measure, called mutation score, is computed considering the number of dead mutants with respect to the number of generated ones. Mutation testing is very efficacious in revealing faults, however, it has a high computational cost related to the execution of mutants. There are several strategies to reduce the mutation testing cost. For instance, a subset of mutants can be randomly selected, or a smaller set of operators can be used, or even search based algorithms can be applied to select the best mutants or operators without reducing the global mutation score. Experiments reported in the literature show that no strategy has been proven to be the best one in all contexts. The determination of the best strategies and their parameter combination is only possible by conducting experiments comparing the strategies in specific scenarios, which can increase the testing cost. The use of an inadequate strategy might compromise the test efficacy and implies in unsatisfactory cost reduction. An automatic approach to select and configure strategies can assist the tester in this task. Hyper-heuristics are viable options for this end, since they are used for selecting or generating heuristics (strategies) instead of trying to solve the problem directly. Hyper-heuristics have been studied in the literature and good results were obtained in the Search Based Software Engineering field. However, few works have investigated the usage of hyper-heuristics in mutation testing, and none of them support the generation or configuration of mutation testing strategies. In light of this, this work proposes a hyper-heuristic based approach called Sentinel. This approach has as main objective the automatic generation of mutant reduction strategies for the cost reduction of mutation testing without compromising the test efficacy in terms of mutation score and ability in revealing faults. Sentinel uses features from several kinds of mutant reduction strategies and combines them to generate new strategies that are potentially more effective. The idea is that the tester executes Sentinel, reuses the automatically generated strategies and consequently does not need to select and configure strategies manually. This eases the mutation testing application and thus reduces the overall cost of the testing activity. For assessing the feasibility of Sentinel, multiple experiments were performed. The experiments were conducted using 10 different systems with 4 versions of each, in a total of 40 systems. The strategies generated by sentinel showed to be significantly more effective than conventional strategies of the literature. Furthermore, they kept their good results in newer versions of the software, which demonstrated their reusability. Keywords: hyper-heuristic, software testing, strategy generation, mutant reduction, grammatical evolution, search based software engineering, multi-objective optimization

Inferring test models from user bug reports using multi-objective search

Bug reports are used by software testers to identify abnormal software behaviour. In this paper, we propose a multi-objective evolutionary approach to automatically generate finite state machines (FSMs) based on bug reports written in natural language, to automatically capture incorrect software behaviour. These FSMs can then be used by testers to both exercise the reported bugs and create tests that can potentially reveal new bugs. The FSM generation is guided by a Multi-Objective Evolutionary Algorithm (MOEA) that simultaneously minimises three objectives: size of the models, number of unrealistic states (over-generalisation), and number of states not covered by the models (under-generalisation). We assess the feasibility of our approach for 10 real-world software programs by exploiting three different MOEAs (NSGA-II, NSGA-III and MOEA/D) and benchmarking them with the baseline tool KLFA. Our results show that KLFA is not practical to be used with real-world software, because it generates models that over generalise software behaviour. Among the three MOEAs, NSGA-II obtained significantly better results than the other two for all 10 programs, detecting a greater number of bugs for 90% of the programs. We also studied the differences in quality and model performance when MOEAs are guided by only two objectives rather than three during the evolution. We found that the use of under-approximation (or over-approximation) and size as objectives generates infeasible solutions. On the other hand, using as objectives over-approximation and under-approximation generates feasible solutions yet still worse than those obtained using all three objectives for 100% of the cases. The size objective acts as a diversity factor. As a consequence, an algorithm guided by all three objectives avoids local optima, controls the size of the models, and makes the results more diverse and closer to the optimal Pareto set